Repository navigation
Conversation
### Background The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy. ### What problem does this PR solve? Issue Number: apache#70251 Related PR: apache#70327 (native reader and catalog read policies) PR Type: Feature Problem Summary: Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add `mongodb_read_preference` and `mongodb_read_concern`, supporting `SET` and `SET_VAR`, and pin their effective values in each scan descriptor. Both default to `inherit`, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations. ### Details - **Change map:** 70 production lines changed, 281 test-source lines changed, 62 generated expectation lines, and 38 documentation lines changed. - **Scope:** Data-read policies for native MongoDB scans. Preference supports `inherit`, `primary`, `primaryPreferred`, `secondary`, `secondaryPreferred`, and `nearest`; concern supports `inherit`, `local`, `majority`, and `available`. Values are case-insensitive. Snapshot and linearizable reads remain unsupported. - **Design:** Register validated, forwarded session variables and resolve them through the existing connector session-property snapshot. Reuse the existing scan descriptor and BE collection policy application. Session overrides do not mutate catalog configuration, metadata clients, or transport pool identity. URI tags, maximum staleness, and hedge options remain effective; an incompatible primary override fails during planning without exposing the URI. - **Usage:** `SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local';` or `SELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ ...`. Set either variable to `inherit` to use its catalog/URI policy again. - **Default behavior:** `inherit` independently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior. - **Compatibility:** No connector SPI, Thrift, BE code, or storage-format changes. Metadata discovery and collection-identity verification retain their existing policies. Read concern does not establish a snapshot shared by scans. ### Testing All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE: - Red/green: the new scan-plan tests first failed because overrides were ignored, and the new session tests first failed because the variables were unregistered; both passed after implementation. - `run-fe-ut.sh --run` with `SessionVariableMongoReadPolicyTest`, `VariableMgrExportTest`, `MongoScanPlanTest`, and `MongoCatalogPropertiesTest`: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding. - `mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test`: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity. - `build.sh --fe -j12`: passed. FE core and MongoDB connector Checkstyle: zero violations. - Generated `test_mongodb_read_policy.out` using the regression runner, checked its results against the synthetic fixture, then independently reran `test_mongodb_native` and `test_mongodb_read_policy`: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery. - Additional replica-set integration: source profiler verified primary/secondary selection and the actual read concern for 8 sequential reads and 32 concurrent reads across 4 isolated sessions. Also verified URI tags with a mode override and policy restoration after a failed hint. This was an additional local acceptance check; the committed SQL suites use the existing standalone fixture. ### Release note Add `mongodb_read_preference` and `mongodb_read_concern` session variables for native MongoDB queries, with statement-level SET_VAR support. Both default to `inherit` to retain the catalog's existing read policy. ### Check List (For Author) - Behavior changed: Yes (see Details; explicit overrides now affect native MongoDB data reads). - Does this need documentation: Yes; the connector README is updated in this PR. - Pre-submission status: Ready. Local checks passed; CI is pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, connector documentation, and this description. - Human review: Maintainer review pending; the agent completed a full diff self-review and local validation.
Gabriel39
requested review from
924060929,
CalvinKirs and
morrySnow
as code owners
November 12, 2026 03:15
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
5 of 13 tasks
Contributor
There was a problem hiding this comment.
Reviewed all eight changed paths at de95c7b. No new actionable findings and no existing blocking inline comments. Review converged after two rounds, including independent session, connector, and risk-focused reviews and a final challenge of this frozen payload.
Critical checkpoint conclusions:
- Goal and evidence: SET and SET_VAR policies independently inherit or override catalog/URI defaults and are pinned in Mongo scan descriptors. New FE and connector unit tests and the regression suite cover the intended behavior; test sources were inspected, not executed.
- Scope and reuse: the change is focused and uses the existing session-property snapshot, scan provider, descriptor, and native reader.
- Concurrency and locks: each scan retains its own session map, descriptor and collection policy; pooled clients are leased exclusively. No new shared mutable policy, lock order or work under metadata locks was introduced.
- Lifecycle: range descriptors are deep-copied, cursor/collection cleanup precedes pool return, and the existing failure path closes acquired resources. SET_VAR restoration runs in the executor's finally path, including planning errors. No new static initialization or resource lifetime was introduced.
- Configuration: both variables default to inherit, validate supported values case-insensitively, update without restart, and participate in generic session export and FE forwarding.
- Compatibility: existing Mongo descriptor fields are reused; no connector SPI, function symbol, storage format or new wire field changes. Missing session keys preserve inheritance. Existing native policy consumers accept the canonical values.
- Thrift/protobuf IDs: no gensrc files changed; branch ID comparison is inapplicable.
- Parallel paths: collection, view and time-series scans use the same provider. Prepared external queries replan, forwarded queries carry the variables, and Mongo tables are excluded from SQL result caching. Metadata discovery and collection identity verification retain independent policies.
- Conditional checks and errors: unsupported policies fail rather than fall back. The primary/URI routing-criteria check matches the native consumer's constraints; inherited criteria remain attached to the scan (checked against the pinned C driver and Java driver implementations).
- Test coverage: inspected defaults, supported and rejected values, case handling, URI precedence, primary conflicts, immutable snapshots, transport identity, hints and restoration. The committed standalone fixture cannot establish replica selection or concurrent replica behavior.
- Expected results: the fixture's four rows have seq values 1 through 4, matching count=4 and sum=10. Variable and hint/restoration expectations match the SQL sequence; SHOW output is ordered and aggregate results have one row. Result generation was not independently reproduced.
- Observability: variables are visible through SHOW/session inspection; existing changed-variable audit capture and native errors remain available. No new metric or high-volume log requirement was identified.
- Transactions and persistence: no source transaction, EditLog protocol or failover behavior changes. Existing reflected session-variable serialization covers the added strings; read concern does not create a shared snapshot across scans.
- Data writes and atomicity: this change only affects external reads; no write, delete-bitmap, rowset or Doris visible-version logic changes.
- FE-to-BE propagation: canonical effective preference/concern values reach the existing TMongoScanDesc on each native range; constant folding and OLAP point-query paths do not read Mongo collections.
- Performance and memory: added resolution does bounded local work per scan, without new I/O or row-loop work. No new long-lived allocation or BE memory ownership was introduced.
- Other concerns and user focus: no additional focus was supplied. Both initial risk items were dismissed with concrete upstream/downstream evidence, and no code-level candidate remains unresolved.
Validation limits: static inspection only. No builds, tests, benchmarks, cluster operations or runtime reproduction were performed. Author-reported replica-set profiling and concurrency checks were not independently verified. These execution limits do not leave an uncovered portion of the required static review.
Review converged after 2 round(s).
Contributor
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The native MongoDB catalog sends read preference and read concern to the BE in each scan descriptor. Today those values come only from catalog properties or the connection URI, while metadata discovery uses its own primary/default policy.
What problem does this PR solve?
Issue Number: #70251
Related PR: #70327 (native reader and catalog read policies)
PR Type: Feature
Problem Summary:
Users cannot choose MongoDB routing and visibility for an individual session or query without changing the shared catalog. Add
mongodb_read_preferenceandmongodb_read_concern, supportingSETandSET_VAR, and pin their effective values in each scan descriptor. Both default toinherit, preserving existing catalog/URI behavior. Most of the diff adds tests and generated regression expectations.Details
inherit,primary,primaryPreferred,secondary,secondaryPreferred, andnearest; concern supportsinherit,local,majority, andavailable. Values are case-insensitive. Snapshot and linearizable reads remain unsupported.SET mongodb_read_preference='secondaryPreferred'; SET mongodb_read_concern='local';orSELECT /*+ SET_VAR(mongodb_read_preference='primary', mongodb_read_concern='majority') */ .... Set either variable toinheritto use its catalog/URI policy again.inheritindependently resolves each policy from URI options first, then catalog defaults. Queries without overrides retain their existing behavior.Testing
All checks below ran locally against MongoDB 8.0.20, with the new FE and the existing ASAN native BE:
run-fe-ut.sh --runwithSessionVariableMongoReadPolicyTest,VariableMgrExportTest,MongoScanPlanTest, andMongoCatalogPropertiesTest: passed. The three new session tests cover defaults, supported/unsupported values, connector snapshots, isolation, and forwarding.mvn -f fe/pom.xml -pl :fe-connector-mongodb -am test: 35 connector tests passed, including URI precedence, independent policy inheritance, incompatible routing criteria, and stable transport configuration identity.build.sh --fe -j12: passed. FE core and MongoDB connector Checkstyle: zero violations.test_mongodb_read_policy.outusing the regression runner, checked its results against the synthetic fixture, then independently rerantest_mongodb_nativeandtest_mongodb_read_policy: 2 suites passed, 0 failed/fatal/skipped. Covers SET/SET_VAR, all supported modes/levels, invalid values, failed-hint restoration, URI policy inheritance, and recovery.Release note
Add
mongodb_read_preferenceandmongodb_read_concernsession variables for native MongoDB queries, with statement-level SET_VAR support. Both default toinheritto retain the catalog's existing read policy.Check List (For Author)
AI Disclosure